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^ IMAGE CONVERSION AMD ENCODING TE CHIMiqiJFg 

y / F'eld of the Invention 

/ The present invention is an Improved technique for deriving depth maps 

from one or more 2D Images. 
5 Background of the Invention 

A number of image processing tasks require that the depth of objects 
within an image be known. Such tasks include the application of special effects 
to film and video sequences and the conversion of 2D images Into stereoscopic 
3D. Detennining the depth of objects may be referred to as the process of 
10 creating a depth map. in a depth map each object is coloured a shade of grey 
such that the shad© indicates the depth of the object from a fixed point. Typically 
an object that is distant will be coloured in a dark shade of grey whilst a close 
Object will be lighter. A standard convention for the creation of depth maps is yet 
to be adopted, and the reverse colouring may be used or different colours may be 
15 used to indicate different depths. For the purposes of explanation in this 
disclosure distant objects will be coloured darker than closer objects, and the 
colouring will typically be grey scale. 

Historically the creation of a depth map from an existing 2D image has 
been undertaken manually. It will be appreciated that an image is merely a series 
20 Of pixels to a computer, whereas a human operator is capable of distinguishing 
objects and their relative depths. 

The creation of depth maps involves a system whereby each object of the 
image to be converted is outlined manually and a depth assigned to the object. 
This process is understandably slow, time consuming and costly. The outlining 
step is usually undertaken using a software program in conjunction with a mouse. 
Examples of a software program that may be used to undertake this task is 
Adobe "After Effects". An operator using After Effects would typically draw 
around the outline of each object that requires a depth to be assigned and then fill 
or "colour in" the object with the desired shades of grey that defines the depth or 
30 distance from the viewer required. This process would then be repeated for each 
object In the image. Further, where a number of images are involved, for 
example a film, it will also be necessary to carry out these steps for each image 
or frame of the film. 
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In the traditional system the outline of the image would typically be 
described some form of curve, for example a Bezier cun^e. The use of such a 
curve enables the operator to alter the shape of the outline such that the outline 
of the object can be accurately aligned with the object. 
5 Should a series of Images require depth mapping e.g., a film or video, then 

the process would be repeated for each frame in the sequence. 

It is likely that the size, position and/or depth of an object may change 
through a sequence. In this case the operator is required to manually track the 
Object .n each frame and processing each frame by correcting the curve and 
_ 1 0 updating the object depth by changing the shade of grey as necessary. It will be 

^ appreciated that this is a slow, tedious, time consuming and expensive process, 

fi Previous attempts have been made to improve this process. The prior art 

H describes techniques that attempt to automatically track the outline of the object 

^ as .t moves from frame to frame. An example of such a technique is the 

■OJ 15 application of Active Contours (ref: Active Contours - Andrew Blake and Michael 
Isard - ISBN 3^540-76217^5). The main limitation of this approach is the need to 
teach the software implementing the technique the expected motion of the object 
being tracked. This is a significant limitation when either the expected motion is 
not known, complex deformations are anticipated, or numerous objects with 
different motion characteristics are required to be tracked simultaneously. 

Point^based tracking approaches have also been used to define the motion 
of outlines. These are popular in editing environments such as Commotion and 
After Effects. However, their application is very limited because it is frequently 
.mpossible to identify a suitable tracking point whose motion reflects the motion of 
the object as a whole. Point tracking is sometimes acceptable when objects are 
undergoing simple translations, but will not handle shape deformations, 
occlusions, or a variety of other common problems. 

An Israeli company. AutoMedla. has produced a software product called 
AutoMasker, This enables an operator to draw the outline of an object and track 
It from frame to frame. The product relies on tracking the colour of an object and 
thus fails When similar coloured objects intersect. The product also has difficulty 
tracking objects that change in size over subsequent frames, for example, as an 
object approaches a viewer or moves forward on the screen. 
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None of these approaches are able to acceptably assign, nor track, depth 
maps, and thus the creating of the depth maps is still a manual s/stem 

Other techniques are described in the prior art and rely on reconstructing 
the movement of the camera originally used to record the 2D sequence The 
f.mitat,on of these techniques is the need for camera motion within the original 
image sequence and the presence of well-defined features within each frame that 
can be used as tracking points. 
Object of the Invention 

Presently, it is necessary for an operator to manually create a depth map 
for each frame of an image, so as to obtain acceptable results. It is an object of 
the present invention to reduce the number of frames that require manual depth 
creation, thereby reducing the time commitments for operators creating the depth 
maps. 

There remains a set of frames for which the depth maps are still to be 
16 created manually. It Is a further object of the invention to assist the manual 
process of depth map creation for these frames. 
Summary of the invention 

With the above objects in mind the present invention provides a method of 
creating a depth map including the steps of: 
20 assigning a depth to at (east one pixel or portion of an image; 

detemiining relative location and image characteristics for' each said at 
least one pixel or portion of said Image; 

utilising said depth(s), image characteristics and respective relative 
location to determine a configuration of a first algorithm to ascertain depth 
25 characteristics as a function of relative location and image characteristics; 

utilising said first algorithm to calculate a depth characteristic for each pixel 
or portion of said image; 

wherein said depth characteristics form a depth map for said image. 
In another aspect the present invention provides a method of creating a 
30 depth map Including the steps of: 

assigning a depth to at least one pixel or portion of an image; 
determining x,y coordinates and image characteristics for' each said at 
least one pixel or portion of said image; 
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u«lisinfl saw dep,h(s), ,„,age charac.eri„rcs and respective x,y ooordinates 

coordinates and Image characteristics; 

usiislng said flist argorithrr, to calculate a depth characerfsllc for each pixel 
5 or portion of said image; ^ 

Wherein said depth characteristics form a depth map for said image 
In a further aspect the present invention provides a method of creating a 
senes of depth maps for an image sequence including the steps of 

receiving a depth map for at least one frame of sard image sequence- 
□ utilis-ng said depth map to determine a configuration of an algorithm to 

g ascertain the depth characteristics as a function of relative position and image 

^2 characteristics; ^ 

m utilising said algorithm to create a depth map for each frame of said image 

5 15 ,n yet a further aspect the present invention provides a method of creating 

^ a senes of depth maps for an image sequence Including the steps of 

selecting at least one key frame from said image sequence- 

for each at least one key frame assigning a depth to at least one pixel or 
portion of each frame; 



□ 



detem.,n,ng relative position (for example x. y coordinates) and image 
charactenstics for each said at least one pixel or portion of each said frame- 

ut,l,sing said depth(s). Image characteristics and relative position for each 
sa,d at least one frame to determine a configuration of an algorithm for each said 
at least one frame to ascertain depth characteristics as a function of relative 
25 position and depth characteristics; 

utilising each configuration of said algorithm to calculate depth 
Characteristics for each pixel or portion of each said at least one frame- 

Wherein said depth characteristics fom, a depth map for each said at least 
one frame. 

^0 utilising each depth map to detemiine a second configuration of a second 

algonthm to ascertain the depth characteristics for each frame as a function of 
relative position and image characteristics; 
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tran,^ """"" "'"^ '^'^^ '^^<^ ™ps for each 

frame of sard image sequence. 

create a "! ""tr' '° ^ ™y In fa.. 

create a number of d,fferen. functions in orter to create the depth maps as a 

5 resu, Of the ,e,a.,Ve position and image cha«c,er,s«ce. ,n the prLrredTylm 
the relafve position will be a measure of the x.y coordinates 

Which flrr ™^ '» 

wh«h frames ,n a sequence are to be considered key ,ra„,es, for example each 

10 r r "'^ « an input ,0 the 

10 algorithm to further refine the processing. P"i 'o ine 

Brief Oesorlptlon of the Invention 

The invention Is Intended to Improve the process of producing depth maps 
for aasocated 2D images. This preferred embodiment invokes t^o phases o 
9enerat,ng key-frame depth maps, and generating me remaining maps 

The first phase obtains a small amount of data from the user This data is 
—a 0, the basic structure of the scene. The ao image and this as^te: 

bllen .m " ''"'"^ ™'«'onshlp 

i 20 r„ T ^= -age characteristics include, enough 

2 ■= f(».y,R,G.B) 
for each pixel in the frame that the user has defined 

in the 1111°*"" ^^'^ '° '^-ining pixels 

o r ' ' " *^ «an refine the,, data 

to mprove the accuracy o, the depth map. „ should be noted that the initial depth 
data need no. necessarl^ be specified by a user - ,t may be detemtined by sol 

mo« nT',H '"^'^ '° "''"^ ^ from 

mot,on algorithm or deriving depth estimates from stereo Images 

The second Phase requires ZD images and associated depth maps to be 
P^i e a. setected keyframes. T.e depth maps a, these keylmes may ^ 
generated for example as previously disclosed by the applicants, or prodld 
automatically using depth capnrre techniques molding, although no, „mlt d tt 
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The 2D image and aasociated dapth map(s), tor each key-frame is 
depm z aaa,gr,ed .o each pixel in .he renialning irames, ite . and y location and 

ioT rIT:'*: '""-^^ -e no, limited 

to, the RGB value of each pixel. In general the algorithm eoives the equation 

2 = '(x.y.R.Q.B) 
for each pixel in the l<ey-(rames. 

10 then preaentedrth each subsequent frame between the 

adiacent key-frames and for each pixel usee the algorithm ,o calculate the value 

In the Drawings 

Figure 1 shows one embodiment of the training process of Phase One 
Rgure 2 shows one embodiment of the conversion process of Phase One " 
Figure 3 shows one embodiment of the training process of Phase Two 
Figure 4 shows one embodiment of the conversion process of Phase Two 
.n«o. " ' '"""'"^ ^'""'^'^ ""-y P^«'«°n the feature 
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Figure 6 Shows an alternate depth map generation process for Phase two 
Figure 7 shows an alternative method to determine depth of an individual 
pixel in Phase two. 

sample?"" ' '"^ °' '^^"''"^ ^^"^'^^'^ *^-ning 

Figure 9 illustrates the calculation of depth from a number of candidate 
training samples. ^a'luiaaxe 

Detailed Description of the Inventton 

The invention provides an improved technique for deriving depth maps 

T ""'^ '"'^^'^'^y PH-es. each 

of Which ideally incorporates an automated learning process. 
Phase One 

The first phase operates on a single image. A user is presented with the 
.mage and defines approximate depths for various regions In the image using a 
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L'i n ' - P^*' The 

r^ul. o, «„3 „ „a, ,he depth is deflned for a s.bse, „ ,he pW, in .he 

.he u.r'T^' " ' "° '^^^ ' '^^ '0 

he user. The use, can .hen assign deplh ,o various pixels wWhln U,e image 2 In 

the example of flgure , ,he pixels marked ■■X" are pixels for which a dep.h has no. 
been specifled by .he user. The sys.em .hen cor,ela.ee .he 2D Image 1 wl.h .he 
depm dale 2 provided by .he user, and utilises a .raining algoritt^m 3 to assls. In 
the creation of a mapping function 4, which Is capable of solving a funcOon for .he 
depth of each pixel in the image. 

.K™' ""^'"'^ ""''"^^ "«'"'"9 tha. is used 

ZZT""'"^ r'"' ''"''"^ '° ~^ ^ ^^P'^ 

P^xel ,n .he sa,d single image. This process may be ln.erac.ive, in that .he user 

may de« no approximate depths for only a few regions. Based on .he results of .he 

learning process for «,e said regions »e user may provide fur^er depth 

I e ..mates for regions where the teaming process performed poorly. This 

h . . . ™' « ''«ge- It Should 

be need .ha. the ini.ial depm data need no. be necessanly be specified by a user 
- 1, may be delemiined by some ofter process as descrtbed above. 
Create Mapping Funcion 

"^^ ^y^'--" P'ovided wl.h .ho image and some pixel depths, the 
system then and analyses the pixels with defined depths in order to create a 
mapping (unction. The mapping function may be a process o, funcion .ha. tekes 
as ,npu. any measuremen. of a pixel or a set of pixels from the image and 
provides as oulput a depth value for the pixel or se. of pixels. 

Individual pixel measurements may consist of red, green and blue values 
or Cher measuremems such as luminance, chrominance, con.«s. and spa«al 
measurements such as horizonttl and vertical posLloning in the ImLge 
Al.erna.,vely the mapping function may operate on higher level image featu^is 
such as larger sets of pixels and measurements on a se. of pixels such as mean 



Li. 



25 



30 



9. AUG. 2001 15:16 ^BeRMARK PATENT 613 98196010 ^ NO. 51 73— P. 9/33. 




8 



^^a sets Of p«e,. may ,o, example r.pr«en. segments in rmage, being 
sets of connected pixels fomiing a homogenous region 

For Illustrative purposes only, a p«el may be mpresented in the fom, 
=» x,y,R,G,B, z 

P«el, R,G,B represent the red, green and blue values cf that pixel and z 
has specified a value. 

10 The mapping function is learnt by capturing the relationship between 

.mage data and depth data for the pixels identified by the user. The mapping 
uncon may take «,e form of any genenc-processing unit, where input dam i! 
reccved, processed, and an output given. Preferably, .his p,x,cessing unit is 
amenable to a teaming process, where its nature Is detennined by examination of 

15 the user data and oonesponding image data. 

outo,,, uT' °' "'^"■"'"'P "^^^ '"P"' *3lred 

-"I- be understood by those who have worfced in the areas of artiflcial 
n,ell,ge„ce or machine learning, and may take on many fom,s. I, is noted that 
these persons would not nom,ally won< in the areas of stereoscopic systems or 

known and rnclude, although are not limited to, neural networks, decision trees 
dec,s,on g«phs, model trees and nearest-neighbour classifiers. Preferred 

25 measurement of mapping error and tha, genera ise 

26 «>tisfactonly for values outside the original data set. 

The learning algorithm may either attempt to detemtine the relationship 
be^een the 2D image information and the depth globally over the whole Image 
or locally over smaller spatial areas. 

30 .n,- ™° ™' ""^'^'^ ^P"' maps for the 

30 entire sequence. 

This can be exemplified by figure 2, which inputs data from the 2D Image 
1, rnto the created mapping function 4, to create a depth map 5 of the 2D Image 
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r ''^ '^^-^ -9oH.hm for learning decision 

For illustrative purposes only, the teaming algorithm may be oonsidered to 

Zn = ka.Xn + kb.yn + Kc.Rn + kd.G„ +ke.Bn 



where 

10 
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n is the nth pixel In the key^frame image 
Zn is the value of the depth assigned to the pixei at x„,y, 
ka to ke are constants and are determined by the algorithm 
Rn is the value of the Red component of the pixel at x^ 
Gn is the value of the Green component of the pixel at Xn y^ 
15 Bn is the value of the Blue component of the pixel at x„,y„' 

This process is illustrated In Figure 1 

It will be appreciated by thcae skilled in the art that the above equation is a 
mpl,,,cat,oo tor purpose, o, explanation only and would not wo.^ idearin 
P^*e. n a praot,ca. implementation using, ,or example, a neural network and 
9^" large number of p^els in an Image, the network would learn one large 

he K-values may vary ac,x,ss different x,y positions In the Image, adapting to 
local rmage features, «ucipiing lo 

Apply Mapping Function to 2D Image 

frame TJ"7n'°" "''"'"^ ^"^'^ « >° 

ft.n« on are determined in a similar manner as that presented to the mapping 

ean^^l by presentrng the measurements of a singte pixel as input, the mapping 
func^on Will now require these same measurements as input. With these 
.he mapprng function performs Its learnt task and outputs a depth measurement 
Ag-n. in the example for a single pixel, this depth measurement may be a simple 
depth value. In this example, the mapping function Is applied across the en^re 
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•mage, to complete a full set of deoth data fh« ■ 

mapping n^ncion was trained J„T " JIT', " 

flan««.e such larger sets o, pllT o 1 " '° 

^ ~n. „ ... ... . , ~: 

10 may b. mada ,o «,e u.er da,a and ,ha process repeated ,o co„eo. these regions 
The ™pp.ng n.no«on ™y also he applied to «her tran^s to generate depth 

I. will be appreciated by those skilled in the art of Machine Learning that 

15 ™s pZcr° r ^ °' 

l^i edTrt " "'"^ and includes, bu, Is no. 

Obit's. In ,h,s embod,men., the process of generalising me .ser data to me 

o..c.the.se.es'.aybe^:;„,zzr:e:::ir 

unohons may be applied to tl» obiects to specify the depth of these ob^l 
hereby constructing a depth map for the Image. These functions may taklTe 
form Of depth ramps and other ways of defining the depth of objects as de ed „ 
25 Ihe AppLoantsphor application POT/AU00/0070O. ^nned ,n 

In a further alternative embodiment, the training algohthm may attempt to 

algomhm ,h,s helps to overcome the difficulty of over-tralnlng. Over-trainlno 
re^rs to the situation where the learning algorithm simply remembers the ZLg 

ir lino ' '° ' - 

w^mou, 3a,n,ng any understanding of the concept of multiplication itself. t^Is 

problem ,s known In the field of machine learning, and an approach to relieving 

.he problem is to Introduce random noise into the training data. A good Jig 
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the date r«fh.r . encouraged to learn the nature of 

ine data rather than s mplv rememher it a„ , 

approach refers to th. n ^"'^"'^'^ embodiment of this 

5 function "'"^ ^'^^ ^^^'"'"^ -'9--*^- the 

Zn = ka-Xn + Kb-Vn + k^-Rn + kd-Gn +ke.B„ 

10 ! H °' "^"^ '^"^"'^ ™"''>^'- 'n tt>e prafe^d 

10 smbodimaninonoteeisaddedtothezcomponsm. 

Learning Process 

fn the prefenred embodimen. ths inputs to the teaming process are- 
inc,udin7dZr " ^"""^ ~ 

LcttilTT °' -^h-oteris^os 

inciudere;"!" :7:::::":' 'rr^ 

20 i.»,n- * ■ • * ^"^ "^P'" °' Pi'-el- Tiie aim of the 

lie r""" ' '° """'^'^ ' °' oia^si^catioTpC 

Whose oharaolerlstics include position (x,y) and colour (RGB) 

inv„, ''l-^m^ation sample, the first stage of the learning algonthm 

~ ,dent„>„ng a subset o, the training samples that share "sirt^iar i^ 
oharactenst,cs to the classification pixels in question. 
25 Searching for Training Candidates 

To identify training samples with similar characteristics to the current 
c^ss ,ca„on sample, we consider an n-dlmensional feature space ln"h" 

each dimens,on repressnflng one of me image characerlsfics: x.y.R G B The 
« axrs of th,s space are normalised » account for differences in the I^ge ;f el^ 
*mens,on. We may therefo. refer ,o the d«erences hetween samp as ul^ 

Dy 10^ (Of me absolute range o„he R component, relative ,o a second sampte 
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classto«r ? """'"^ '"^'^ <«««noe from .h. 

ne classtotlon sample and ,s used in the calculation of depth, Distance In the 

Mahalanob,s d,stance motrico are used to provide better «,ults Alternative 
means o, stretching the range of the data such aa histogram e,.2t™ o 
10 pnncpal component analysis of the RGB Yuv or h<!u ™ 

similar benefits. ®^ components provide 

and is " ' o' ^epth 

spatial or temporal autocorrelation the radius Is set to smaller values than for 
15 ,mages with low spatial or temporal autocorrelation 

soac. 1^ ''"^ <" «>e feature 

space^ For example, the search radius in the x-axls may be dWerent from the 
radius in the axis representing the ted colour intensity. pZlr 1 

a spatr^dts 7:rT ' "° "^"^^ '^"^ ^^"^P'- - 



20 



Rgure 8 illustrates a simplified example of M,e candidate sea«;hlna 

sTarx T T ^""^ ' ' -.aJsT:: 

Itl ; °' "'""^ ''-'-«°™ '"e red colour 

.ans„y for the purposes of illustration. Within this space are a numL o 
ra n,ng samples 20. Within a distance of a nrs, radius 21 of the taZ pL 
he.e are no training samples. The learning process therefor! Zd t 

30 :rat:tr:i:~ ^ - - -- 3 

Alternative search strategies may be used to Identify suitable tralnino 
« d.a es^ such strangles training data Is stored In struc.:es such as a h^ 
tree, K-d or an n-d,menslonal Vorono, diagram. Although such strategies 
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Distance Waighled Learnlnj 

To calculate a depth for any given classification sample we require cne or 
™. t air^ng samples which a. deemed ,o be simltar .o me Cass Jicn samp^^ 
'0 as descnbed above Wg rpffir * • ■ °"">iadmpie 
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Target Pixel Depth ^ (wrD1+w2*D2+w3*D3) 

(W1+W2+W3) 



target pixel and D3 r, h h . ^"'"^ ^ 

^ s."a ~;:::r '^^^^ - ~ - 

Alternative Embodiment 

^" f smbodlmenl the learning process analyses the complete 

into a ser'7r" t ^'"^ ^"'"^^ - P^^IUoned 

Ziple In ,h ' "'~ ^ -P^-n-a^on of this 

me target pixaM ibased on which region if occupies 

.he J. ~ ^"^'^ •» P^rfor. the partition of 




10 



9. AUG. 2001 15:1 8 ^ERMARK PATENT 613 98196010 -^0.5173 — P. 16/33- 



15 

Phase TWo 

10 Create Mapping Function 

This mapping ,s learnt by capturing the relationship between the L .rami i! 
data and depth map data available for «,ose imaflee 

Where rputZ'"" '^'^ *^ <" ~proces.ing unit, 

pt^n: n:::z:ror"^' 

W anamination "thTC^ r^ndT' """" 
fi«.H . ®' corresponding depth man in th*^ 

^« ot machine learning, euch mapping i^ncions are nown a jini e 
2= are not limited to, neura, networks, decision trees, dedsil rlt 
model trees and nearest-neighbour classifiers ^ ^ ' 

desired'lTZ'TT "''""""'^^ '"^^ and 

imT™ ? ' P™'*"'' "o-" the 2D key-frame 

.mage s presented to tbe training algorithm, 77,1, mfonnation may be pllnt™ 
on a p„el by pixel basis, „l«re pixel measurements are provided such as t 
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■eve, ,™«e .1 X//™ "™ 

For Illustrative purposes onlv the 2n im=«« u 
form ""^^ represented in the 

x,y,R.G,B 

to the training algorithm However if hioherlev^^iim.^ * / Presented 
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Of the aepm ,0, ,ha. set of pixels, such as mean and variance' 
^^^^ For i„os.ra.,ve purposes only, the depth .ap n,ax be .presented ,n the 
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Where x and y represent the x and y coordinates of each pixel and z re„r« , 
the depth value assigned ,0 tha, corresponding pixel 

The process of learning this relationship between Input data and d.., h 

OutDut would hf^ iinw«..«»^ *. . "'f^ui aaia, ana desired 

some ' "^PP'^f '"""to" «Woh minimises 

some measurement of mapping error, minimises 

O^eJllT!"' '""^^ '° ^'^'^^ ^'-'ton^Hips between 

*e 2D-image Intonnatlon and the depth map present in the key-frame axam^Ls 

sequence. Examples of successful learning algorimms known in the art are the 
beck-propagation algorithm lor learning neural netwo.<s, the C4.5 algorith: Z 




9. AUG. 2001 15:18 ^PeRMARK PATENT 61 3 981 9601 0 NO. 5173 — P. 18/33- 




17 



where 

n Is the nth pixel In the key-frame Image 
is the value of the depth assigned to the pixel at x„.y„ 

^ to k. are constants and are determined by the algorithm 
y Rn .s the value of the Red component of the pixel at x. y„ 

:j G„ ,s the value of the Green component of the pixel at x„ y„ 

fw ^" "^'^^ the Blue component of the pixel at x„ y„' 

S '* ^PP^clated by those skilled in the art that th« 

largo number of pixels in .n im^ge, ,he ne^vo^lc would team on^ lar«t , 
contaimns mayo values, n,u,Bp,ioa,io„s and addidons '^^ 

20 Apply Mapping Function 

Tha invention nex, take, ,h,s ™pp,„g 
Of 2D .mages tha, do no, ye, have depm n,ap, available. For a given 2DTaaet 
2 "-^ lnpu,s .0 ,be ™pping ,u™=.io„ are de,e„n,ned ,n a iZr " Zr 

pixel as input, the mapping function will now reauire the.,^ 
frtr+H«-- 1 . . '''sse same measurement"! 

30 exal,rl '"^^ ''^ ' '^P* value. In ,his 

sxampie, ,ne ™pp,ng ft,nc,ion is applied ac^se the enUre image sequence to 
oo.p e,e a ,u„ ee, o, dep,b data tor the i™ge sequence. .^ZIZ^^Z 

gene,a,e such targer se,s o, pixels ,or the new in^age. The higher-leve. 
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measurements on these sets of oixels ^^r^ m.n 
- .an,. ™nn. . ... ^ ^Z:!^ Zl^ 

estabtished the maoDina f..r..f- . Process. With these inputs now 

.^e. oo.e„, , - - 

commonality, in the simplest ca.;p t«,n i,^ * 

.0 .he number o, tey-frames ,h3, ™y be used TZ ! 
Further, ,he™ is no res.riCon ,o .he nuLr Tnn Jn- """""" 

separated by one or „ ° J^' ''7'^'^'™^ ^-^<'^'"'-' -wo key frames, 
eecond pMse Of prlssZ t'--' as inpu,e ,o .h,s 

key frames firet Fr«m». * u Processing frames closest in time to the 

y ii^mes rirst. Frames that have been processpH th«r, k 
depth map subsequent frames '^^"^^^ 

variable. It Is possible that the d.nfh • ^ ^^'^"^ ^ 

eacho.er L.::::rrrnrrirrrrr:::r^^^^^^ 
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become more inclined to r.^. ■ '^ey-frame. the training algorithm will 

generated is associated with the source fr,™ i • . 

the ir .rz::: .r r r '-'^^^ — 

Key ,ran,e. T^is process is iiiustrated n 1 k 7 k' °" 

1 1 -e compared ,o the closes, soul k^ I Jo e 2 " ' 
«ra.e in the i™9e sequence (step " andTsT Tho ila" 
that described praviously uses a .p/k J ' 
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30 



and oaltrit"":' f " P^^' -«P*= 0. 

~«.hedi:rzrcurr::r:^^^^^^^^^^ 

-^^...nce to the secor. Key trame is ther, the depth o. 
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20 

1 



cfeprA= ^^^^^ — *ni.. - 



□ ^0 



Where Dl and D2 are the depth calculated from key frame 1 .nd u 
frame 2, respectively. ^ ^ '^^^ 

In some cases, the (earning process cannot determine a depth value for . 
9.ven pixel. If during the above calculation process one of IT I 
.epth estimates could not be determined then' th; ar^^^^^^^^^ 

r re^a^^^^^^^^^^^ ~ - - — ^ '-^ - ~ 
Proce. is ™ • — and the 



y It should be noted that only one key frame is necessan. to generate d.nth 

ry 15 maps for any other frame. However. In situations where the de^^^^^^ 

^ - iallyaffectingthenat:::^^^^^^^^ 

may beTrn'olildT";' ' " ^^^^ ^-"-9 stage 

:rp:ranr~ 



25 



at any pixel of an Image in the sequence. 
This process is Illustrated in Figure 4. 



30 



^^^^^ 

a,, n=„ .ay.ran,a ,„«p.. ,n .he sequence. This may be axp,o,.a. aHn 
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encoding of this depth information. It is expected th.t tn 

be transmitted with a relative, smai, ZToTZ ZTr' "^^^ 

Significant compression Of the depth informal ^ 

~ li:::^^^^^ apart in the 

across the t^^^^^^^^ ~ ^ " ^" ^^''^^ 

can be wHtten to a .HI «1T^^ ^'^'^^ ^-^«on 

worth of depth irrr^^^rif r h ^^'"^ 
300 hx^o , Effecfvely. this represents a file size of 6000 / 20 ^ 

300 bytes per frame. In a practical implementation the effective col!! " 
be substantial. effective compression will 



10 

be substantial, 
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30 



:;irr ir.^^^^^^^^^ - ~ 

5 compatible 3D imaae q.nr. th 2D-,mage source j.e, a 2D 

.ho addition 0, 3D ,nfor™„on to 

small overhead. ^""^""^ "'^'•^"^ """^ a very 
AotC: or . real. 

,0 .hoZiras ::irt:,r r ^^^^ 
.pes o, r„app,„« ...lons :T.7.rr:xz:z'7'^T 

process or .uIiZI d I ' " ""'"^ "'^ 

n,a,.e.ra„.rd , l , Z:nr;'- |,°"" '"^^'''"^ 
su.«e , or readme con jronr ' ^'^^ 

'^»e.™.ro,::x:::— ^^^^^^^ 
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depth maps oaloulatea. Since ,he key-franies «pr«em a s^all fraction of the 
.0.^ number Of frame., this new ,echn,<,.e rep..en,s a aub„an.,a. ImprovCn 
■n converaion efficiency both in temis Of time and cost 

-o^^e^c^atton of depth maps for other than ,he product^ of etereolpic 

It Will be known to those skilled in the art that depth maps are used 
ex^en ,ve,y within the special effects industry in a process called rotoscoplng . 
orde to composite live ao«on, cr computer gene^ted images, within a 2D ir^age 

am r '° """""^ "'"'""^ ' "^"^ ^-^^ - -'^ 

ram o 2D image. These mattes enable the add.iona, images to be c»r,posed 

Imaoe 3" '° ■'^'^ ^""""^ 2== 

-age. The ,nven.,on described above enables the rapk, generaUon of such 
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man to h T "^"^'"P^"' ^^'^ - depth 

map ,0 be obta,ned from a live scene. Typically these use laser range finding 

scl^iques and are genericaily known as LIDAR de>.ces. In orde, to oaptu e 
rZl™r r''" ^" — « ~ ' 

L DAR dev,ces to be constructed that need only capture depth maps a. a train 
Of the v,deo field rate, or other infrequent periods, and the missing depth m^ 
produced by interpolation using the teohnk,ues described in this Invention 



